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FOREWORD 

The  Defense  Communications  Engineering  Center  (DCEC)  Technical  Notes 
(TN's)  are  published  to  inform  interested  members  of  the  defense  community 
regarding  technical  activities  of  the  Center,  completed  and  in  progress. 
They  are  intended  to  stimulate  thinking  and  encourage  information 
exchange;  but  they  do  not  represent  an  approved  position  or  policy  of 
DCEC,  and  should  not  be  used  as  authoritative  guidance  for  related 
planning  and/or  further  action. 

Comments  or  technical  inquiries  concerning  this  document  are  welcome, 
and  should  be  directed  to: 

Di rector 

Defense  Communications  Engineering  Center 
1860  Wiehle  Avenue 
Reston,  Virginia  22090 
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I.  INTRODUCTION 


1 . PURPOSE 


The  purpose  of  the  wideband  test  and  evaluation  was  to  determine  and 
compare  the  performance  of  certain  wideband  speech  coders.  Wideband  refers 
to  a transmission  data  rate  of  16  kb/s  or  greater.  It  was  desired  to 
evaluate  the  performance  of  recently  developed  wideband  speech  coders 
and  compare  them  with  speech  coders  based  on  the  continuously  variable 
slope  delta  (CVSD)  algorithm.  Improved  wideband  speech  coders  have 
potential  application  to  the  Defense  Comnuni cations  System.  The  test 
conditions  represented  a wide  range  of  operating  environments  similar 
to  those  of  the  Narrowband  Consortium  T&E  [1].  Speech  coder  performance 
was  measured  with  respect  to  speech  quality  and  intelligibility.  The  overall 
performance  of  the  speech  coders  was  compared  and  the  sensitivity  of  the 
speech  coders  to  various  operating  conditions  was  evaluated. 


2.  SCOPE 


A complete  set  of  tests  was  performed  for  seven  different  speech  coders 
Each  coder’ was  an  implementation  of  a specific  speech  coding  technique. 
These  algorithms  are  continuously  variable  slope  delta  (CVSD)  modulation, 
adaptive  delta  modulation  (ADM),  adaptive  residual  coding  (ARC),  and 
adaptive  predictive  coding  (APC).  In  the  following  sections  each  device  is 
briefly  described.  • 

a.  CVSD-B  COD  16.  This  CVSD  speech  coder  was  designed  and  built  by 
Codex  Corp!  The  algorithm  was  designed  to  produce  optimal  speech  quality 
under  conditions  of  bit  error  rates  less  than  5%.  The  operating  data  rate 
was  16  kb/s.  A detailed  description  of  this  device  is  contained  in  the 
Codex  final  report  [2].  An  analysis  of  the  performance  sensitivity  of 

the  CVSD  speech  coder  with  respect  to  the  algorithm  parameters  is  presented 
in  the  report  [3]. 

b.  COD  ARC  16.  This  speech  coder  was  designed  and  built  by  Codex 
Corp.  The  algorithm  is  based  on  adaptive  residual  coding  (ARC).  The 
operating  data  rate  is  16  kb/s.  A detailed  description  is  contained 
in  [2]. 

c.  COD  ADM  16.  This  speech  coder,  which  was  also  designed  and 

built  by  Codex  Corp. , Is  based  on  the  adaptive  delta  modulation  (ADM) 
algorithm.  The  operating  data  rate  is  16  kb/s.  A detailed  description 
is  also. contained  in  [2].  • 

d.  SCHIL  16/3.  This  CVSD  speech  coder  Is  based  on  the  so  called 
Schilling  algorithm.  The  operating  data  rate  is  16  kb/s.  The  bandwidth 
of  the  audio  output  filter  is  approximately  3 kHz. 


e.  SCHIL  32/4.  This  device  is  similar  to  the  coder  described  above 

i except  that  the  operating  rate  is  32  kb/s  and  the  audio  bandwidth  is  4 kHz. 

f.  SYLM  APCQ  16.  This  speech  coder  was  implemented  on  the  Sylvania 
Programmable  Signal  Processor.  The  coding  algorithm  was  designed  by 

GTE  Sylvania  and  is  based  on  APC  with  an  adaptive  multilevel  quantizer. 

The  operating  data  rate  is  16  kb/s.  A detailed  description  of  this  speech 
» coder  is  contained  in  the  Sylvania  final  report  [4], 

g.  LOG  CVSD  16.  This  device  is  based  on  the  CVSD  algorithm  and  was 
built  by  General  Dynamics.  The  operating  data  rate  is  16  kb/s. 

3.  TEST  CONDITIONS 

» 

The  speech  coders  were  evaluated  with  respect  to  simulated  operational 
conditions.  These  conditions  are  divided  into  the  following  groups: 

a.  Optimal  Condition.  This  condition  is  intended  to  represent  an 
ideal  operating  environment  with  an  input  speech  signal  having  a high 
signal  to  noise  ratio  and  low  distortion  level.  In  addition,  there 
were  no  transmission  bit  errors  and  an  attempt  was  made  to  adjust  the 
audio  input  level  to  an  optimal  setting  for  each  speech  coder. 

b.  Acoustic  Background  Noise.  Test  tapes  were  used  which  contained 

( speech  data  recorded  under  conditions  of  acoustic  background  noise. 

These  tests  included  three  acoustic  background  noise  conditions:  (a)  office 
noise,  (b)  airborne  command  post  (ABCP)  noise,  and  (c)  helicopter  noise. 

c.  Transmission  Bit  Errors.  This  test  condition  consisted  of 
injecting  bit  errors  into  the  transmission  data  stream.  The  probability 

, of  the  occurrence  of  a bit  error  is  based  on  a normal  distribution. 

Bit  error  rates  of  1%  and  0.1%  were  used.  Bit  errors  were  produced  by 
an  error  generator  that  produced  a pseudo-random  binary  sequence. 

d.  Dynamic  Range.  The  audio  input  level  of  the  speech  coder  was 
adjusted  in  order  to  measure  the  effects  on  the  performance  of  the 

i coder.  Measurements  were  made  to  determine  the  optimal  audio  level 

of  each  speech  coder  and  this  was  defined  as  the  reference  level  of  0 dB. 

To  test  dynamic  range,  performance  was  evaluated  for  audio  input  levels 
of  +6  dB,  -12  dB,  and  -20  dB. 

e.  Tandem  Operation.  This  test  condition  was  designed  to  measure 
the  performance  of  speech  coders  in  multiple  link  configurations  and  in 
tandem  with  a narrowband  speech  coder.  Multiple  link  tandem  tests  on  two, 
three,  and  five-link  configurations  were  performed.  The  operation  of 
each  speech  coder  in  tandem  with  a linear  predictive  coder  (LPC)  was 
tested.  The  LPG  speech  coder  was  operating  at  a data  rate  of  2.4  kb/s. 


4.  MODEM  TESTING 


I 


1 


A set  of  tests  were  conducted  In  order  to  evaluate  effects  of  processing 
MODEM  link  signals  with  a CVSD  speech  coder  operating  at  32  kb/s.  The 
results  of  this  testing  are  discussed  in  Appendix  A. 
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II.  BACKGROUND 


1 . NARROWBAND  CONSORTIUM  T&E 

In  May  1976,  the  final  report  of  the  Narrowband  Digital  Voice 
Consortium  was  published.  The  objective  of  the  consortium  was  to 
recomnend  a narrowband  speech  algorithm  that  could  be  adopted  to  satisfy 
operational  needs  of  DoD  for  secure  voice  on  narrowband  transmission 
media  such  as  HF  radio,  switched  telephone  networks,  and  satellites. 
Equipment  from  the  following  generic  classes  of  voice  processors  designed 
for  narrowband  channels  were  tested  and  evaluated: 

o Adaptive  Residual  Coder  (ARC) 

o Adaptive  Predictive  Coder  (APC) 

o Linear  Predictive  Coder  (LPC) 

o Channel  Vocoder  (CV) 

o Hybrid  Speech  Coder  (TRIVOX). 

The  test  program  was  managed  by  the  T&E  Subcommittee  under  the  chair- 
manship of  DCA  and  was  for  the  most  part  performed  at  the  DCA  Hybrid 
Simulation  Facility  with  NRL  performing  the  HF  and  simulated  satellite 
test.  Performance  evaluation  was  performed  using  both  intelligibility 
and  quality  tests.  Intelligibility  was  measured  using  the  Diagnostic 
Rhyme  Test  (DRT)  and  the  Consonant  Recognition  Test  (CRT).  Quality  was 
measured  using  the  Paired  Acceptance  Rating  (PAR).  The  narrowband  voice 
processor  operated  at  data  rates  from  2.4  kb/s  to  9.6  kb/s.  The  LPC-10 
algorithm  was  recomnended  In  the  Consortium  final  report  as  the  DoD 
standard  for  narrowband  speech  processing. 

2.  PERFORMANCE  EVALUATION 

The  wideband  speech  coder  performance  evaluations  were  based  on 
Intelligibility  and  quality  tests.  The  Diagnostic  Rhyme  Test  (DRT) 
has  been  found  to  be  a reliable  procedure  for  measuring  word  intelli- 
gibility [5].  The  DRT  was  used  for  the  wideband  evaluations,  and  these 
scores  can  be  compared  with  the  results  of  the  Narrowband  Consortium 
T&E  [1].  The  DRT  does  not  necessarily  give  a valid  measure  of  the  overall 
performance  of  a speech  coder.  Specifically,  high  DRT  scores  do  not 
necessarily  Imply  high  user  acceptance.  For  this  reason  speech  quality 
evaluations  were  also  conducted. 

The  purpose  of  the  quality  evaluations  was  to  determine  the  overall 
user  acceptance  of  the  speech  quality  of  each  coder.  The  Diagnostic 
Acceptability  Measure  (DAM)  was  the  procedure  selected  to  accomplish  the 
quality  evaluations  [6].  The  DAM  Is  an  evaluation  method  designed  to 
eliminate  many  of  the  inconsistencies  associated  with  the  Paired  Accept- 
ability Rating  Method  (PARM)  that  was  used  in  the  Narrowband  Consortium 
T&E.  Both  the  DAM  and  DRT  employ  listeners  who  record  subjective  responses 
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for  evaluation.  Therefore,  there  Is  a significant  degree  of  error 
associated  with  each  method;  the  DRT  scores  generally  have  a smaller 
standard  error  than  the  DAM  scores. 

Evaluation  results  should  be  Interpreted  carefully  with  the  knowledge 
that  user  preference  Is  difficult  to  quantify.  Corresponding  to  each 
DAM  score  Is  an  equivalent  PARM  score  so  that  these  results  can  be  compared 
with  the  Narrowband  Consortium  T&E.  Figure  I is  a plot  that  shows  the 
estimated  relationship  between  DAM  scores  and  user  acceptance. 

3.  TEST  PROCEDURES 

The  speech  data  that  was  evaluated  according  to  the  intelligibility 
and  quality  tests  was  stored  on  audio  magnetic  tape.  These  tapes  were 
prepared  in  the  same  laboratory  facility  used  for  the  Narrowband  Consortium 
T&E  [1].  A description  of  the  test  facilities  is  contained  in  the  Consortium 
Final  Report.  As  in  the  Narrowband  Consortium  T&E,  the  magnetic  tapes 
were  given  to  Dynastat  Corp  for  intelligibility  and  quality  evaluations. 


III.  TECHNICAL  DISCUSSION 
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1 . COMPARISONS  OF  SPEECH  CODER  PERFORMANCE 

The  results  of  the  intelligibility  and  quality  tests  were  analyzed 
to  determine  the  statistical  significance  of  differences  in  system  DRT 
and  DAM  scores.  As  in  the  Narrowband  Consortium  T&E,  the  Neuman-Kuels 
test  [7]  was  used  to  rank  systems  and  estimate  the  statistical  significance 
of  system  scores.  The  results  are  grouped  according  to  test  conditions. 

The  test  results  are  displayed  in  the  form  of  a table  containing  a 
matrix  which  is  used  to  compare  a set  of  speech  coders  under  a given 
operating  condition.  Alongside  the  matrix  are  listed  the  speech  coder 
mean  score,  standard  error,  names,  and  identification  numbers.  Above 
the  matrix  is  shown  the  system  identification  numbers.  The  systems  are 
ranked  in  order  of  decreasing  test  scores.  The  system  rank  order  corresponds 
to  the  matrix  row  number.  The  numbers  in  the  matrix  above  the  main 
diagonal  are  the  differences  between  system  mean  scores;  the  (i,j)  element, 
where  j is  greater  than  i,  is  the  difference  in  mean  scores  between  the 
system  ranked  i and  the  system  ranked  j.  For  example,  in  Table  II,  the 
number  1.6,  row  2 and  column  4 is  the  difference  between  the  score 
93.4,  of  the  system  ranked  number  2 (CVSD  ARC  16). 
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The  Neuman-Kuels  test  is  a systematic  procedure  for  determining  the 
statistical  significance  in  the  difference  between  these  mean  scores. 

Below  the  main  diagonal  are  numbers  that  represent  the  confidence  levels 
of  the  difference  between  system  mean  scores.  For  example,  a confidence 
level  of  0.99  implies  that  the  probability  is  0.01  or  less  that  the  system 
scores  are  samples  from  the  same  probability  distribution.  Therefore, 
the  confidence  level  is  0.99  that  the  difference  in  scores  is  statistically 
significant.  The  (i.j.,)  element,  where  i is  greater  than  j,  is  the  confi- 
dence level  corresponding  to  the  difference  In  scores  between  the  system 
ranked  i and  the  system  ranked  j.  The  scores  were  evaluated  for  three 
confidence  levels,  0.99,  0.95,  and  0.90;  no  entry  in  the  matrix  Indicates 
that  the  confidence  level  is  less  than  0.90.  For  example,  in  Table  II, 
the  confidence  level  of  the  difference  in  mean  scores  between  COD  ARC  16 
and  SCHIL  32/4  is  0.99;  between  COD  ARC  16  and  CVSD  B COD  16  it  is  0.90; 
and  between  COD  ARC  16  and  SYLM  APCQ  16  the  confidence  level  is  less 
than  0.90.  This  implies  that  there  is  not  a high  level  of  confidence  that 
the  mean  scores  of  CVSD  B COD  16  and  SYLM  APCQ  16  are  significantly 
different. 

a.  Optimal  Conditions.  Table  II  shows  the  DRT  scores  and  statistical 
test  results  corresponding  to  six  male  speakers  under  optimal  conditions. 

As  noted  previously,  optimal  conditions  correspond  to  a quiet  acoustic 
background  with  no  transmission  bit  errors.  As  expected,  the  test  results 
show  that  the  performance  of  SCHIL  32/4  was  better  than  all  the  other 
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speech  coders  operating  at  16  kb/s.  It  can  also  be  seen  that  this 
superior  performance  is  statistically  significant.  The  LOG  CVSD  16 
was  significantly  poorer  than  any  other  technique  on  this  test.  Based 
on  the  performance  of  this  technique  under  the  other  conditions  tested 
and  on  the  DAM  score  for  optimal  condition,  this  score  is  15  points  lower 
than  expected.  This  low  score  Is  attributed  to  a misadjustment  in  the 
input  audio  level  which  is  particularly  critical  for  this  technique. 

Lower  than  expected  DRT  scores  also  occurred  with  LOG  CVSD  16  for  test 
conditions  of  1%  BER  and  LPC  into  LOG  CVSD  16. 

It  can  be  seen  from  Table  II  that  the  results  of  the  Neuman-Kuels 
analysis  can  be  used  to  group  the  test  results  in  terms  of  statistical 
significance.  Examination  of  the  first  column  of  confidence  levels  In 
Table  II  indicates  that  the  difference  between  the  score  achieved  by 
SCHIL  32/4  and  the  other  techniques  is  great  enough  to  indicate  a true 
difference  in  performance.  Examination  of  the  second  column  in  Table  II 
indicates  that  the  difference  between  the  score  achieved  by  CVSD  B COD  16 
and  SYLM  APCQ  16  is  not  great  enough  to  indicate  a true  difference  in 
performance  (first  entry  in  the  column  is  blank).  Differences  in  score 
between  CVSD  B COD  16  and  the  remaining  techniques,  however,  are  great 
enough  to  indicate  a true  difference  in  performance  as  Indicated  by  the 
presence  of  the  confidence  level  scores.  The  statistical  significance 
of  the  differences  in  the  scores  for  the  remaining  techniques  can  be 
determined  by  examining  the  remaining  columns  in  a similar  fashion. 

Table  III  shows  the  DAM  results  corresponding  to  two  male  speakers 
under  optimal  conditions.  The  correspondence  between  the  DAM  scores 
and  estimated  user  acceptance  can  be  found  in  Figure  1.  It  can  be  seen 
that  SCHIL  32/4  had  the  highest  overall  score  and  SYLM  APCQ  16  had  the 
best  score  among  the  16  kb/s  coders. 

Table  IV  contains  the  DRT  results  corresponding  to  three  female 
speakers.  CVSD  B COD  16  and  SYLM  APCQ  16  had  the  best  DRT  scores  among 
the  16  kb/s  coders.  Table  V contains  the  DAM  results  corresponding 
to  one  female  speaker.  It  can  be  seen  that  based  on  this  test  the  DAM 
scores  are  divided  into  two  groups.  SYLM  APCQ  16  and  SCHIL  32/4  are 
In  the  first  group  and  the  remaining  coders  are  in  the  second  group. 

In  suimiary,  under  optimal  test  conditions  the  Sylvania  APC  algorithm 
(SYLM  APCQ  16)  had  the  best  overall  performance  of  the  16  kb/s  algorithms. 

b.  Background  Noise  Conditions 

(1)  Office  Noise.  Table  VI  contains  the  DRT  results  for  one 
male  speaker  under  conditions  of  an  office  noise  environment.  It  can 
be  seen  from  these  results  that  several  16  kb/s  coders  performed  as 
well  as  SCHIL  32/4.  Table  VII  contains  the  DAM  results  corresponding 
to  three  male  speakers. 
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Table  VIII  contains  the  DRT  results  corresponding  to  one  female  speaker. 

In  this  test  COD  ADM  16  scored  as  well  as  SCHIL  32/4.  In  Table  IX  the 
DAM  results  corresponding  to  one  female  speaker  are  given. 

(2)  Airborne  Command  Post  Noise.  The  DRT  and  DAM  results 
for  airborne  command  post  background  noise  are  shown  in  Tables  X and  XI 
respectively.  These  results  are  based  on  male  speakers,  and  it  can  be 
seen  that  the  scores  are  generally  several  points  lower  than  the  scores 
for  office  background  noise. 

(3)  Helicopter  Noise.  Table  XII  contains  the  DRT  results  for 
male  speakers  under  conditions  of  helicopter  background  noise.  It  can 
be  seen  that  the  DRT  scores  are  considerably  lower  than  those  under 
optimal  conditions.  In  addition,  the  statistical  test  showed  that 
there  is  no  significant  difference  between  any  of  the  systems  including 
SCHIL  32/4.  Table  XIII  contains  the  results  of  the  voice  quality  analysis 
for  three  male  speakers.  These  scores  are  quite  low,  indicating  a very 
small  percentage  of  user  acceptance.  Also,  it  can  be  seen  that  there  is 
no  significant  difference  in  DAM  scores  except  for  CVSD  B COD  16  which 
had  the  lowest  score. 

c.  Transmission  Bit  Errors.  Tests  performed  under  conditions  of 
transmission  bit  errors  are  valuable  because  the  corresponding  results 
indicate  the  expected  speech  coder  performance  on  actual  transmission 
media.  Most  transmission  media  degradations  can  be  shown  to  correspond 
to  certain  transmission  bit  errors.  These  bit  errors  can  be  divided 
into  two  categories,  burst  errors  and  normally  distributed  random  errors. 

The  test  results  of  this  report  are  based  on  random  errors  derived  from 
a normal  probability  distribution. 

(1)  BER  of  0.1%.  The  performance  of  speech  coders  such  as 
CVSD  whose  design  is  based  on  delta  modulation  does  not  degrade 
significantly  under  conditions  of  relatively  high  transmission  bit 
error  rates.  Table  XIV  shows  the  DRT  results  for  male  speakers 
corresponding  to  a BER  of  0.1%.  For  all  the  speech  coders,  except 
LOG  CVSD  16,  there  is  only  a small  difference  in  DRT  scores  compared 

with  optimal  conditions.  Table  XV  contains  the  DAM  results  for  male  speakers. 
These  results  are  also  similar  to  those  for  the  optimal  conditions. 

Tables  XVI  and  XVII  contain  the  DRT  and  DAM  results  respectively  for 
female  speakers.  These  results  indicate  that  SYLM  APCQ  16  performed 
better  than  all  other  16  kb/s  speech  coders  under  this  BER  condition. 

(2)  BER  of  1%.  Table  XVIII  contains  the  DRT  results  for  male 
speakers  under  conditions  of  1%  BER.  The  CVSD  B COD  16  performed  as 
well  as  SCHIL  32/4  in  this  test.  Table  XIX  contains  the  speech  quality 
test  results  for  male  speakers.  Tables  XX  and  XXI  contain  the  DRT 

and  DAM  results  respectively  for  one  female  speaker.  The  DAM  results 
indicated  no  statistically  significant  difference  between  systems  for 
the  female  speaker. 


(1)  6 dB  Input  Level.  Speech  intelligibility  and  quality  tests 
were  conducted  to  measure  the  performance  sensitivity  of  the  speech  coders 
with  respect  to  audio  input  levels.  In  a realistic  operating  environment 
audio  input  levels  will  not  necessarily  be  adjusted  for  optimal  performance 
due  to  such  factors  as  variability  of  speakers  and  microphones.  Consequently, 

It  Is  useful  to  have  an  Indication  of  the  performance  sensitivity  to 

audio  levels.  Table  XXII  contains  the  DRT  results  corresponding  to  one 
male  speaker  and  an  audio  Input  level  6 dB  above  the  estimated  optimal  level. 
Because  these  results  represent  one  male  speaker,  it  is  not  meaningful  to 
compare  them  with  the  results  of  Table  II  which  represent  six  male  speakers. 
However,  the  large  difference  in  the  DRT  score  o*  LOG  CVSD  16  indicates 
that  the  audio  level  for  the  optimal  condition  test  was  not  set  properly. 
Generally  the  DRT  scores  are  high  with  no  statistically  significant 
difference  between  systems  except  for  SCHIL  16/3  which  was  ordered  with 
the  lowest  score.  The  speech  quality  test  results  for  male  and  female 
speakers  are  shown  in  Tables  XXIII  and  XXIV  respectively.  In  these  tests 
the  user  preference  for  SYLM  APCQ  16  is  approximately  the  same  as  for  SCHIL  32/4. 

(2)  -12  dB  Input  Level.  Table  XXV  shows  the  DRT  results  for 
one  male  speaker  corresponding  to  an  audio  input  level  12  dB  below  the 
estimated  optimal  level.  Table  XXVI  shows  the  DAM  results  for  male 
speakers.  In  this  speech  quality  test  SYLM  APCQ  16  and  CVSD  B COD  16 
performed  as  well  as  SCHIL  32/4.  In  Table  XXVII  It  can  be  seen  that  for 
one  female  speaker  SYLM  APCQ  16  performed  better  than  all  other  speech 
coders  with  respect  to  speech  quality. 


(3)  -20  dB  Input  Level.  Table  XXVII  contains  the  DRT  results 
for  one  male  speaker  under  conditions  of  an  input  level  20  dB  below  the 
estimated  optimal  level.  Tables  XXIX  and  XXX  contain  the  DAM  results 
for  male  and  female  speakers  respectively.  It  can  be  seen  that  on  the 
speech  quality  tests  the  performance  of  SYLM  APCQ  16  is  superior  to  all 
other  speech  coders. 

e.  Tandem  Configurations.  Tests  were  conducted  to  determine  the 
performance  of  speecn  coders  in  certain  tandem  configurations.  A single 
transmission  link  is  defined  as  two  Identical  speech  coders  connected 
together  such  that  there  Is  an  analog  to  digital  conversion  followed  by  a 
digital  to  analog  conversion.  Multiple  tandem  links  are  defined  as 
several  single  links  of  Identical  speech  coders  connected  in  series.  A 
wideband-narrowband  tandem  is  the  series  connection  of  a single  wideband 
link  and  a narrowband  link. 

(1)  Multiple  Links.  Tests  of  multiple  link  tandems  were 
conducted  for  two,  three,  and  five  links.  Tables  XXXI  and  XXXII  contain 
the  DRT  and  DAM  results  respectively  for  male  speakers  under  the  condition 
of  two  links  In  tandem.  For  female  speakers  the  DRT  and  DAM  results  are 
shown  In  Tables  XXXIII  and  XXXIV  respectively. 
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The  DRT  and  DAM  results  are  shown  in  Tables  XXXV  and  XXXVI 
respectively  for  male  speakers  under  the  condition  of  three  links  in 
tandem.  Table  XXXVII  contains  the  DAM  results  corresponding  to  one 
female  speaker.  It  can  be  seen  that  the  performance  of  SYLM  APCQ  16 
was  the  lowest  of  all  speech  coders  tested. 

Table- XXXVIII  contains  the  DRT  results  corresponding  to  five  links 
In  tandem  for  one  male  speaker.  In  this  test  the  performance  of  CVSD  B 
COD  16  was  superior  to  the  other  16  kb/s  speech  coders.  Tables  XXXIX 
and  XL  show  the  results  of  the  speech  quality  tests  for  male  and  female 
speakers  respectively.  It  can  be  seen  that  speech  quality  Is  degraded 
significantly  under  conditions  of  five  links  in  tandem. 

(2)  Wideband-Narrowband  Tandem.  The  full-duplex  tandem  operation 
of  wideband  and  narrowband  speech  coders  was  evaluated.  The  narrowband 
terminal  was  a linear  predictive  coder  (LPC)  using  ten  predictor 
coefficients  and  operating  at  a data  rate  of  2.4  kb/s.  A detailed 
description  of  this  narrowband  speech  coder  is  given  in  reference  [8]  . 

Tables  XLI  and  XLII  show  the  DRT  and  DAM  results  respectively  for  male 
speakers  corresponding  to  LPC  tandemed  with  the  wideband  speech  coders. 

These  results  indicate  no  significant  difference  in  performance  between 
speech  coders  except  for  LOG  CVSD  16.  Tables  XLI I I and  XLIV  contain  the 
DRT  and  DAM  results  respectively  for  female  speakers.  These  results 
also  indicate  no  significant  difference  in  the  performance  of  the  speech 
coders  except  for  COD  ADM  16. 

The  intelligibility  and  quality  results  shown  in  Tables  XLV  and 
XLVI  respectively  represent  the  performance  of  the  wideband  speech  coders  • 
operating  into  LPC  with  male  speakers.  The  DRT  and  DAM  results  for 
female  speakers  are  shown  in  Tables  XLVII  and  XLVI I I respectively. 

Overall,  the  best  performance  of  the  16  kb/s  speech  coders  was  achieved 
by  SYLM  APCQ  16. 

f.  Sunmary  of  Results.  The  results  of  the  statistical  analyses  show 
clearly  that  the  Schilling  CVSD  coder  operating  at  32  kb/s  is  superior 
In  performance  to  the  other  speech  coders  operating  at  16  kb/s.  Certainly 
this  result  Is  expected.  In  order  to  compare  the  performances  of  the  16  kb/s 
speech  coders,  the  ranking  of  each  speech  coder  for  each  operating  condition 
Is  presented  in  Table  XLIX.  This  table  shows  the  relative  ranking 
corresponding  to  Intel! iglbllity  tests  for  male  speakers.  The  ranking  is 
based  on  the  results  of  the  Neuman-Kuels  analyses.  System  scores  are 
grouped  according  to  the  0,90  confidence  level  and  the  groups  are  then 
ordered.  For.  example,  under  optimal  conditions  In  Table  XLIX,  both 
SYLM  APCQ  16  and  CVSD  COO  16  are  ranked  In  the  first  group.  For  the  male 
DRT,  the  Codex  CVSD  speech  coder  has  the  best  scores  over  almost  all 
conditions.  The  Syl vanla  APCQ  ranks  very  high  except  for  the  multiple 
link  tandems.  Table  L lists  the  ranking  of  the  16  kb/s  speech  coders 
according  to  DAM  scores  for  male  speakers.  Both  SYLM  APCQ  16  and  CVSD  B 
COO  16  are  generally  superior  to  the  other  speech  coders  under  most 
conditions.  Table  LI  lists  the  ranking  of  speech  coders  based  on  DRT 
scores  for  female  speakers.  These  results  indicate  that  SYLM  APCQ  16 
Is  generally  superior  In  performance  to  the  other  speech  coders.  The 
ranking  of  systems  according  to  DAtf  scores  for  female  speakers  is  shown 
In  Table  LII.  These  results  also  Indicate  that  SYLM  APCQ  16  Is  generally 
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superior  to  the  other  speech  coders. 

Certain  operating  conditions  are  associated  primarily  with  the 
tactical  communications  environment.  These  are  helicopter  and  airborne 
command  post  background  noise,  high  bit  error  rate,  and  certain  tandem 
connections.  Under  the  nontactical  conditions,  the  Sylvania  APCQ 
algorithm  provides  the  best  performance  of  the  speech  coders  operating 
at  16  kb/s.  Under  the  tactical  conditions,  the  Codex  CVSD  speech  coder 
provides  the  best  performance  overall. 

2.  INTELLIGIBILITY  AND  QUALITY  TEST  SCORES 

The  individual  intelligibility  and  quality  test  scores  are  presented 
In  this  section  so  that  the  performance  of  each  speech  coder  can  be 
evaluated  under  all  test  conditions.  These  test  results  indicate  how 
the  performance  of  each  speech  coder  is  affected  by  certain  operating 
conditions  and  speakers.  These  results  should  not  be  used  exclusively 
to  compare  the  performance  of  different  speech  coders.  The  statistical 
analyses  described  in  the  previous  section  were  performed  in  order  to 
make  such  comparisons. 

a.  Codex  CVSD  B.  Table  II  lists  the  DRT  scores  obtained  for  the 
Codex  CVSD  speech  coder  operating  under  the  conditions  shown.  The 
Initials  of  the  male  speakers  are  LL,  RH,  CH,  PK,  JE,  and  BV;  the  initials 
of  the  female  speakers  are  JS,  LS,  and  MP.  The  standard  error  of  each 
DRT  is  shown  to  the  right  of  the  score.  For  some  conditions  more  than 
one  DRT  has  been  performed,  and  all  scores  are  listed  in  the  table. 

These  results  show  that  the  Codex  CVSD  speech  coder  produces  highly 
Intelligible  speech  under  almost  all  operating  conditions.  Table  LIV 
lists  the  quality  scores  resulting  from  the  DAM  test.  These  results 
indicate  that  the  user  acceptance  of  the  Codex  CVSD  is  generally  high. 
However,  some  inconsistencies  in  the  DAM  test  are  revealed  in  these 
results.  For  example,  the  scores  under  0.1%  BER  are  much  lower  than  under 
the  optimal  conditions,  although  it  is  known  that  the  effect  of  0.1% 

BER  is  barely  perceptible  compared  with  optimal  conditions. 

b.  Codex  ARC.  The  DRT  scores  of  the  ARC  speech  coder  are  shown 
in  Table  LV.  These  scores  indicate  that  the  ARC  algorithm  generally 
produces  highly  intelligible  speech.  However,  the  scores  under  the 
optimal  conditions  indicate  that  this  speech  coder  is  rather  speaker 
sensitive;  the  DRT  scores -range  from  94.1  to  87.4.  The  DAM  scores  of 
the  ARC  speech  coder  are  shown  in  Table  LVI. 

c.  Codex  ADM.  The  DRT  scores  of  the  Codex  ADM  speech  coder  are 
shown  in  Table  LVI I . These  scores  Indicate  that  the  speech  produced 
by  this  coder  is  highly  Intelligible.  The  DRT  scores  under  the  optimal 
conditions  indicate  that  this  speech  coder  is  speaker  sensitive;  the 
DRT  scores  range  from  91.9  to  84.8.  The  speech  quality  test  scores 
are  presented  in  Table  LVI I I . 
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d.  Schilling  CVSD  16  kb/s.  The  DRT  scores  of  the  Schilling  CVSD 
speech  coder  operating  at  16  kb/s  are  shown  in  Table  LIX.  Additional 
intelligibility  tests  were  performed  with  cutoff  frequencies  of  2.0  kHz 
and  2.5  kHz  for  the  audio  output  filter.  These  scores  indicate  that 
this  coder  generally  produces  highly  intelligible  speech.  The  DAM  scores 
of  this  speech  coder  are  shown  in  Table  LX. 

e.  Schilling  CVSD  32  kb/s.  The  DRT  scores  of  the  Schilling  CVSD 
speech  coder  operating  at  32  kb/s  are  shown  in  Table  LXI.  These  results 
indicate  that  the  intelligibility  of  speech  produced  by  this  coder  is 
quite  high.  The  speech  quality  scores  are  shown  in  Table  LXI I . 

f.  Syl vania  APCQ.  Table  LXI I I lists  the  DRT  scores  of  the  Sylvania 
APCQ  speech  coder.  The  intelligibility  scores  are  generally  quite  high. 
It  can  be  seen  that  speech  intelligibility  is  reduced  significantly  under 
conditions  of  5%  BER  and  five  links  in  tandem.  The  quality  scores  are 
shown  in  Table  LXIV.  The  DAM  scores  are  generally  consistent  with  the 
DRT  scores. 

g.  LOG  CVSD.  Tables  LXV,  LXVI,  and  LXVII  list  the  DRT  scores  of 
LOG  CVSD  operating  at  16  kb/s,  LOG  CVSD  operating  at  32  kb/s,  and  General 
Dynamics  CVSD  respectively.  The  DRT  scores  under  optimal  conditions  are 
10  to  15  points  lower  then  expected.  This  appears  to  be  the  result  of 
•improper  tape  recording  levels  or  malfunctioning  recording  equipment. 
Tables  LXVIII,  LXIX,  and  LXX  list  the  DAM  results  of  LOG  CVSD  operating 
at  16  kb/s,  LOG  CVSD  operating  at  32  kb/s,  and  General  Dynamics  CVSD 
respectively.  The  quality  test  scores  appear  to  be  valid  and  should  give 
a reliable  estimate  of  user  acceptance  of  these  speech  coders. 


IV.  CONCLUSIONS 


Generally,  under  optimal  conditions,  all  coders  produced  speech  of 
high  intelligibility.  The  performance  of  the  CVSD  coder  operating  at 
32  kb/s  was  superior  to  the  performance  of  the  other  coders  operating 
at  16  kb/s.  Overall,  the  performance  of  the  Codex  CVSD  coder  was  superior 
to  that  of  the  other  CVSD  speech  coders  operating  at  16  kb/s.  Under 
nontactical  conditions  the  performance  of  the  Sylvania  APCQ  speech  coder 
was  superior  to  the  CVSD  operating  at  32  kb/s  on  some  tests.  Under 
tactical  conditions  the  performance  of  the  Codex  CVSD  coder  was 
superior  to  that  of  the  other  16  kb/s  speech  coders. 
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TABLE  III.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  OX  INPUT:  OdB  SPEAKERS:  2 MALE 


BELOW  DIAGONAL  » CONFIDENCE  LEVELS 


TABLE  V.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  0*  INPUT:  OdB  SPEAKERS:  1 FEMALE 

SYSTEM 
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TABLE  IX.  1 NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  OFFICE  BER:  0%  INPUT:  OdB  SPEAKERS:  1 FEMALE 


TABLE  X.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 

NOISE:  A3 CP  BER:  OX  INPUT:  0d3  SPEAKERS:  2 MALE 


BELOW  DIAGONAL  » CONFIDENCE  LEVELS 


TABLE  XI.  NEUMAN-WJELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  ABCP  BER:  OX  INPUT:  OdB  SPEAKERS:  3 MALE 
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TABLE  XIII.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  HELI  BER:  0%  INPUT:  OdB  SPEAKERS:  3 MALE 
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TABLE  XIV.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 

NOISE:  QUIET  BER:  .1%  INPUT:  OdB  SPEAKERS:  2 MALE 


ABOVE  DIAGONAL  » SCORE  DIFFERENCES 

BELOW  DIAGONAL  « CONFIDENCE  LEVELS 


TABLE  XV.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  .1%  INPUT:  OdB  SPEAKERS:  2 MALE 
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TABLE  XVI.  NEUMAN-KUEIS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 

NOISE:  QUIET  BER:  .1*  INPUT  OdB  SPEAKERS  1 FEMALE 


BELOW  DIAGONAL  » CONFIDENCE  LEVELS 


TABLE  XIX.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  1*  INPUT:  OdB  SPEAKERS:  2 MALE 


TABLE  XX.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 

NOISE:  QUIET  BER:  1*  INPUT:  OdB  SPEAKERS:  1 FEMALE 
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BELOW  DIAGONAL  = CONFIDENCE  LEVELS 


TABLE  XXI,  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  1%  INPUT:  OdB  SPEAKERS:  I FEMALE 


BELOW  DIAGONAL  - CONFIDENCE  LEVELS 
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TA3LE  XXVI.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  0%  INPUT:  -1 2dB  SPEAKERS:  2 MALE 
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TA3LE  XXVII.  NEUMAN-KUELS  ANALYSIS  i)F  SYSTEM  DAM  SCORES  WITH 

NOISE:  QUIET  BER:  INPUT:  -12dB  SPEAKERS:  1 FEMALE 


BELOW  01 AGONAL  = CONFIDENCE  LEVELS 
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TABLE  XXXIII.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 

NOISE:  TANDEM  2X  j BER:  0%  INPUT:  OdB  SPEAKERS:  1 FEMALE 
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TABLE  XXXIV.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  TANDEM  2X  BER:  03!  INPUT:  OdB  SPEAKERS:  1 FEMALE 


ABOVE  DIAGONAL  * SCORE  DIFFERENCES 

BELOW  DIAGONAL  = CONFIDENCE  LEVELS 


TABLE  XXXV.  NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  ORT  SCORES  WITH 

NOISE:  TANDEM  3X  BER:  OdB  INPUT:  OdB  SPEAKERS:  1 MALE 


ABOVE  DIAGONAL  » SCORE  DIFFERENCES 

BELOW  DIAGONAL  * CONFIDENCE  LEVELS 


TABLE  XXXVI'.  NEUMAN- KUELS  ANALYSIS  OF  SYSTEM  DAM  SCORES  WITH 

NOISE:  TANDEM  3X  BER:  02  INPUT:  OdB  SPEAKERS:  2 MALE 
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TABLE  XL  1 1 1 . NEUMAN-KUELS  ANALYSIS  OF  SYSTEM  DRT  SCORES  WITH 
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*3.0  kHz  output  filter. 
**2.5  kHz  output  filter. 
***2.  kHz  output  filter. 
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APPENDIX  A 


MODEM  Performance  Tests 

1.  Introduction.  A set  of  tests  was  conducted  in  order  to  evaluate  the 
effects  of  processing  MODEM  line  signals  with  the  CVSD  speech  coder  operating 
at  32  kb/s.  Figure  A. I shows  the  test  configuration  with  the  Codex  9600 
MODEM.  A bit  error  rate  tester  (BERT  901)  was  used  to  measure  bit  errors. 

2.  Test  Conditions.  The  Schilling  CVSD  Speech  coder  operating  at  32  kb/s 
was  used  to  process  the  MODEM  line  signals.  Tests  were  conducted  using 
different  audio  bandwidths  of  the  CVSD  coder.  Tests  were  also  conducted 
under  conditions  of  transmission  bit  errors.  The  Codex  MODEM  was  tested 
for  three  data  rates,  4.8  kb/s,  7.2  kb/s,  and  9.6  kb/s. 

2.  Performance  Measurements.  The  MODEM  performance  was  measured  according 
to  bit  errors.  In  each  test  the  number  of  bit  errors  and  the  number  of 
f block  errors  were  determined.  A block  error  is  defined  as  a specified 

number  of  bit  errors  per  block  of  data.  In  addition,  the  MODEM  indicator 
light  showed  the  operating  condition  of  the  MODEM. 

4.  4.8  kb/s  Test  Results.  Table  A. I shows  the  measured  bit  error  rates 
and  block  error  rates  for  the  Codex  4800  b/s  MODEM.  The  error  rates  are 

9 given  corresponding  to  different  transmission  bit  error  rates  produced 

by  an  error  generator  that  introduces  bit  errors  into  the  CVSD  transmission 
data  stream.  The  cutoff  frequency  of  the  CVSD  audio  filter  was  varied 
from  2.0  to  6.0  kHz.  A block  error  is  defined  as  3 bit  errors  within  a 
data  block  of  1000  bits.  MODEM  performance  was  satisfactory  for  transmission 
bit  error  rates  of  10"4  and  less.  However,  performance  was  marginal  for  a 
9 transmission  BER  of  10*3.  In  this  case  the  block  error  rate  was  0.137 

and  the  MODEM  indicator  light  showed  a marginal  operating  condition. 

The  results  indicate  that  the  MODEM  performance  is  insensitive  to  the 
different  audio  bandwidths  of  the  particular  audio  filter  tested. 

5.  7.2  kb/s  MODEM.  Table  A. II  shows  the  measured  bit  error  and  block 
error  rates  of  the  Codex  MODEM  operating  at  a data  rate  of  7.2  kb/s. 

These  measurements  were  made  under  conditions  of  two  different  CVSD 
transmission  error  rates.  The  performance  of  the  MODEM  was  marginal. 

» 6.  9.6  kb/s  MODEM.  Table  A. Ill  shows  the  measured  bit  error  rates  of 

the  Codex  MODEM  operating  at  9.6  kb/s.  The  performance  of  the  MODEM 
operating  at  this  rate  was  unsatisfactory. 

7.  Sunmary.  For  satisfactory  processing  of  the  Codex  4.8  kb/s  modem 
line  signals,  the  data  rate  of  the  CVSD  coder  must  be  at  least  32  kb/s. 
f The  bit  error  rate  of  the  CVSD  data  link  should  be  less  than  10"4. 

Operating  the  modem  at  7.2  kb/s  resulted  in  a marginal  to  unsatisfactory 
link.  Operating  the  modem  at  9.6  kb/s  resulted  in  an  unsatisfactory  link. 
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FIGURE  A. I MODEM  TEST  CONFIGURATION 
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TABLE  A. II.  7.2  kb/s  TEST  RESULTS 


TABLE  A. III.  9.6  kb/s  TEST  RESULTS 
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